build: Extend tokenizer capabilities#1114
Merged
msluszniak merged 1 commit intomainfrom May 6, 2026
Merged
Conversation
12 tasks
78b5a13 to
f1341d2
Compare
msluszniak
reviewed
May 4, 2026
Collaborator
ill take a look tomorrow |
Collaborator
|
is there any particular tokenizer this should be tested with? |
Member
Yes, unigram. You can test it by running model from this PR: #1115 |
mkopcins
approved these changes
May 6, 2026
msluszniak
added a commit
that referenced
this pull request
May 7, 2026
…#1115) ## Description Adds the `paraphrase-multilingual-MiniLM-L12-v2` sentence-transformer model — the second multilingual embeddings model after distiluse, completing #945. Ships **only the XNNPACK 8da4w variant** under `MODEL_REGISTRY.ALL_MODELS` (see "Why a single variant" below). 384-d output, max 126 tokens, 50+ languages. Tokenizer is Unigram + Precompiled normalizer + Metaspace decoder — **requires the bumped `pytorch/extension/llm/tokenizers` runtime from #1114**, so this PR blocks on that landing first and should be rebased onto main once #1114 merges. HF repo: [software-mansion/react-native-executorch-paraphrase-multilingual-MiniLM-L12-v2](https://huggingface.co/software-mansion/react-native-executorch-paraphrase-multilingual-MiniLM-L12-v2) (`v0.9.0` tag, layout mirrors distiluse). **Why a single variant** — TLDR 8da4w works faster then all and was also one of the smallest, without loss in precision. Longer answer: unlike distiluse, where Core ML fp32 won iPhone thanks to ANE acceleration, benchmarks on iPhone 17 Pro + OnePlus 12 (~80-token input, 50 measured forwards after 3 warmups) showed the XNNPACK 8da4w variant Pareto-dominates the other three on both platforms: faster than XNNPACK fp32, Core ML fp32 *and* Core ML fp16 on iPhone, and ~36% smaller steady-state memory footprint than the next-best variant. Likely cause: paraphrase-multilingual-MiniLM-L12-v2 is a smaller model (~118 M params, 12 layers) where Core ML's runtime doesn't push enough work onto ANE for the precision-conversion overhead to pay off. fp16 being slower than fp32 on Core ML for this model is a tell that the runtime is falling back to slower compute units. Shipping only `_8DA4W` keeps the public surface aligned with the data; if a future Core ML or model update flips the verdict, easy to add the other variants back. **Memory methodology note** — the new paraphrase row in `docs/docs/02-benchmarks/memory-usage.md` reports RSS / `phys_footprint` deltas from a clean app baseline (loaded − idle), captured on-device at the same conceptual point. The existing distiluse rows there (36 / 44 MB) come from an older measurement pass with a different (and not reconstructable from the diff) methodology, so the two rows are not directly comparable. A separate pass to re-measure distiluse and other rows with the same methodology would be a good follow-up. ### Introduces a breaking change? - [ ] Yes - [x] No ### Type of change - [ ] Bug fix (change which fixes an issue) - [x] New feature (change which adds functionality) - [ ] Documentation update (improves or adds clarity to existing documentation) - [ ] Other (chores, tests, code style improvements etc.) ### Tested on - [x] iOS - [x] Android ### Testing instructions 1. `cd apps/text-embeddings && npx expo run:ios` (or `run:android`). 2. Pick **"Multilingual Paraphrase (8da4w)"** in the model picker. 3. Add a sentence in one language, query with an aligned sentence in another (e.g. Polish "Słoneczko" against "It's so sunny outside!"). The cross-lingual pair should top the matches. ### Related issues Closes the paraphrase-multilingual half of #945 (the distiluse half landed in #1098). ### Checklist - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have updated the documentation accordingly - [x] My changes generate no new warnings ### Additional notes Blocks on #1114.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR introduces rebuilt binaries that contain new, updated tokenizers.
This iteration features support for more tokenisation models (i.e. unigram, worldlevel) as well as bunch of previously unsupported pre-tokenisers, decoders, post-processors.
Introduces a breaking change?
Type of change
Tested on
Testing instructions
Before merging, test all demo applications. See if all models that proved problematic during bumps in the past are working (i.e. kokoro, multi-method models)
Check all LLM models, see if output is working.
Screenshots
Related issues
Checklist
Additional notes